Disable TLS certificate verification in crawler allows MITM/SSRF#2052
Disable TLS certificate verification in crawler allows MITM/SSRF#2052omnipotentchaos wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Addresses Issue #1747 by hardening the crawler’s HTTP client configuration to prevent MITM risk from disabled TLS verification and to mitigate redirect-based SSRF by disallowing cross-host redirects.
Changes:
- Adds a new
tls_verifysetting (env:SECUSCAN_TLS_VERIFY) and documents it in.env.example. - Updates
crawl_target()to useverify=tls_verifyand disables automatic redirects in favor of an explicit same-host redirect policy. - Adds new unit tests covering TLS verification usage and redirect allow/block behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
backend/secuscan/crawler.py |
Enforces TLS verification via config and replaces follow_redirects=True with a same-host redirect loop. |
backend/secuscan/config.py |
Introduces tls_verify setting to control crawler TLS verification behavior. |
.env.example |
Documents SECUSCAN_TLS_VERIFY for configuration discoverability. |
testing/backend/unit/test_crawler_security.py |
Adds unit tests for TLS verify wiring and cross-host redirect blocking policy. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
utksh1
added
level:advanced
utksh1
left a comment
utksh1
left a comment
There was a problem hiding this comment.
Please narrow and document the redirect policy before merge. The branch combines configurable TLS verification with a new same-host redirect policy; add coverage for relative redirects, redirect-loop/max-hop handling, and the intended behavior when TLS verification is explicitly disabled. Rebase on current main afterward.
utksh1
left a comment
utksh1
left a comment
There was a problem hiding this comment.
TLS verification changes are security-sensitive and this branch is behind main. Please rebase and add coverage for the secure default, an explicit opt-out only where documented, and failed verification without changing crawler request semantics beyond TLS handling.
b7562c2 to
5fd1196
Compare
… test_saved_views
|
@utksh1 Please check now |
Description
This PR resolves a security vulnerability in
crawler.pywherehttpx.AsyncClientwas configured withverify=Falseandfollow_redirects=True, allowing Man-in-the-Middle (MITM) attacks and redirect-based Server-Side Request Forgery (SSRF) to arbitrary internal/external endpoints.Key changes:
tls_verify: bool = Trueconfiguration setting (SECUSCAN_TLS_VERIFYenv var) inconfig.pyand documented it in.env.example.httpx.AsyncClientincrawler.pyto enforce TLS verification (verify=tls_verify).response.historytracking for valid same-host redirect chains.Related Issues
Fixes #1747
Type of Change
How Has This Been Tested?
Ran backend unit tests covering the crawler security policy and helper functions: